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Abstract. Collaborative learning has been shown to be beneficial for older stu¬ 
dents, hut there has not been much research to show if these results transfer to 
elementary school students. In addition, collaborative and individual modes of 
instruction may be better for acquiring different types of knowledge. Collabora¬ 
tive Intelligent Tutoring Systems (ITS) provide a platform that may be able to 
provide both the cognitive and collaborative support that students need. This 
paper presents a study comparing collaborative and individual methods while 
receiving instruction on either procedural or conceptual knowledge. The collab¬ 
orative groups had the same learning gains as the individual groups in both the 
procedural and conceptual learning conditions but were able to do so with fewer 
problems. This work indicates that by embedding collaboration scripts in ITSs, 
collaborative learning can be an effective instructional method even with young 
children. 
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1 Introduction 

While collaborative learning has been shown to be beneficial for both face-to-face 
and Computer Supported Collaborative Learning (CSCL) [9], [14], collaborative 
learning often puts challenges on students and teachers that make it hard to implement 
in the classroom. The challenges teachers face include preparing materials, teaching 
the students collaboration skills, and learning how to manage small groups [3]. For 
students, fruitful collaboration does not happen spontaneously, and collaboration 
scripts are used to support students in their learning [6]. It is important for a script to 
match the learning goals of the activity and to provide enough support for the students 
without over-scripting. Collaboration can be supported through different features such 
as roles, cognitive group awareness, and the distribution of information. The chal¬ 
lenges faced by both the students and teachers can make the use of collaboration 
daunting. Some prior research has indicated that Intelligent Tutoring Systems (ITSs) 
can be a practical way of addressing the challenges of using collaboration in the class¬ 
room. Most CSCL environments are missing the cognitive support that can be benefi¬ 
cial to student learning. An ITS can provide the cognitive support (i.e. step-by-step 



guidance and hint features) that a student needs for collaboration to be successful 
[18], but does not provide support for effective collaboration. The current research 
investigates if embedding a collaboration script into an ITS so it has both the collabo¬ 
rative and cognitive support can help a student to learn successfully. 

Even though collaborative learning has been shown to be successful in some in¬ 
stances, few studies have investigated whether CSCL can have a positive impact on 
learning with young children. The implementation and support of collaboration in the 
classroom is particularly difficult for students in elementary school and may explain 
why there is less research with this age group. An important question then is if col¬ 
laborative learning can be an effective instructional method to use with elementary 
school students and if it would lead to similar learning gains as students working in¬ 
dividually. Some studies have shown successful use of collaboration with elementary 
school students as well, but have either compared the use of a CSCL setting to face- 
to-face collaborative learning (i.e., not supported by computers) without comparing it 
to individual learning or have focused on interventions that mix individual and col¬ 
laborative learning tasks without looking at each separately [1], [8], [16]. Although 
this research has shown positive impacts of young children working in small groups 
and with computers, it is still unknown how the use of a CSCL environment impacts 
the learning outcomes of young children compared to learning individually. This pa¬ 
per aims to address this question through an ITS designed specifically to support col¬ 
laborative learning of children in elementary school. ITSs have been shown to have 
positive impacts on students in this age group when working individually to learn 
fractions [12]. We now extend this research by testing whether a tutor that supports 
collaboration can be effective for learning fractions by elementary school students. 

Although most prior work on ITSs has focused on individual learning, there has 
been some work on combining ITSs with collaborative learning that has shown prom¬ 
ise for supporting learning with high school students [17]. Walker et al. found that 
students working with an ITS redesigned to support collaboration (specifically, peer 
tutoring) had learning gains at least equivalent to those working individually. 

In creating a collaborative tutor, it may be important to consider the possibility that 
individual and collaborative learning activities may be better for acquiring different 
types of knowledge, such as conceptual and procedural knowledge [10]. Conceptual 
knowledge is the implicit and explicit understanding of the principles in a domain and 
how they are interrelated [13]. Procedural knowledge is the ability to be able to per¬ 
form the steps and actions in sequence to solve a problem [13]. Mullins, Rummel, and 
Spada found that with 9 th graders doing algebra, students who worked collaboratively 
on conceptual tasks outperformed those who worked individually and students who 
worked individually on procedural tasks outperformed those who worked collabora¬ 
tively [10]. Again, this study was implemented with older students and the question 
still remains if the same difference will be seen with elementary school students. 

Why would it be better to acquire different types of knowledge through different 
instructional methods? following the Knowledge-Learning Instruction (KLI) frame¬ 
work, instruction should be designed for both the domain and for the type of 
knowledge component to be learned [5]. Simpler instructional methods tend to be 
associated with simpler knowledge components, more complex methods with more 



complex knowledge components. Thus, collaboration, a more complex instructional 
method, would be better for more complex knowledge components, where elaboration 
and a deeper understanding is needed, such as those in conceptual knowledge. More 
specifically, collaborative learning may be successful because the students give and 
receive explanations and construct knowledge through their discussions [4]. On the 
other hand, individual learning would be more geared towards procedural learning 
where practice and repetition are more important for developing fluency. 

In our study, we address the feasibility of using a collaborative ITS with elemen¬ 
tary school students learning fractions. Wc hypothesize that students working collabo- 
ratively will show learning gains on both procedural and conceptual fractions tasks. 
Also, we hypothesize that on conceptual tasks, students working collaboratively will 
have stronger learning gains than students working individually. By contrast, for stu¬ 
dents doing procedural tasks, we hypothesize that those working individually will 
have stronger learning gains than those working collaboratively. These hypotheses are 
consistent with both the KLI framework and the Mullins et al. findings. 


2 Methods 

2.1 Tutor Design 

Informed by our prior work on the Fractions Tutor [12], we developed a new ITS for 
a challenging topic in fractions, learning equivalent fractions. Specifically, we built 
two parallel versions of this tutor for use in our study, one with embedded collabora¬ 
tion scripts and one for individual learning. Both versions had procedural and concep¬ 
tual problem sets. Both were built with CTAT, which we extended to support collabo¬ 
rative tutors [11]. The collaborative ITS combines the cognitive support normally 
provided by an ITS (step-level guidance for problem solving) with embedded collabo¬ 
rative scripts for each tutor problem. The collaboration is supported through the use of 
a shared problem view, roles, cognitive group awareness, and unique information. 
First, the collaborative tutors support synchronous, networked collaboration, in which 
collaborating students sit at their own computer and have a shared (though differenti¬ 
ated) view of the problem state. They can discuss the activity through audio chat. 

Second, the embedded scripts define roles to distribute the activities between the 
students. The roles provide guidance to the students about what they should be doing 
to interact with their partner and help to scaffold this interaction. Students were as¬ 
signed to either a helper role or a problem solver role for each task in a problem. The 
students were informed of their role assignment through the use of icons displayed on 
the interface (see Figure 1). An “ask” icon next to a problem step signaled to the stu¬ 
dent that they were in the helper role and responsible for asking questions and making 
sure both they and their partner understood the answer. A “do” icon next to a compo¬ 
nent meant the student was in the problem solver role and responsible for carrying out 
the step to move the problem solution forward (Figure 1). 

A third collaborative support feature we used in the collaborative problem sets is 
cognitive group awareness. Cognitive group awareness means that group members 



Equivalent Fractions 


Let's make equivalent fractions. 


This is the unit of the fractions. 
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numerator and denominator of the purple fraction needs 
to be multiplied by to get the new fraction to the right of 
the equals sign. 


H " 2 0 






Students see icons displayed next 
to each component informing them 
of their role. 


® Make a fraction where the numerator 
— and denominator are 4 times larger 
:han the purple fraction. 

Make a fraction where the numerator 
® and denominator are 5 times larger 
| ^ |than the purple fraction. 




Students are asked to problem solve 
for a step that uses information their 
partner has solved. 


Fig. l. A collaborative procedural problem: multiplying to make equivalent fractions. 


have information about other group members’ knowledge, information, or opinions 
and has been shown to be effective for the collaboration process [7]. We implemented 
cognitive group awareness by a design pattern in which the collaborative tutor poses a 
question to both students and asks each student to answer independently first without 
being tutored (bottom of Figure 2). After both students answer the question inde¬ 
pendently, the tutor shows them each other’s answers and gives them the opportunity 
to answer the question as a group, which is tutored. This activity allowed each student 
an opportunity to express an opinion and gave each dyad an opportunity to discuss 
and explain their answer choices, especially important when they disagreed. 

The last collaborative support feature is the use of unique information to create a 
sense of individual accountability, a popular feature in scripts such as the jigsaw [2]. 
Individual accountability means that each group member takes responsibility for the 
group reaching its goal [14], By providing each student with information that their 
partner does not have and that is needed to complete the problem, both students have 
a stake in completing the problem. In our problem sets, unique information was im¬ 
plemented by providing one member of the dyad with some information the other 
student did not see. The student would know they had unique information because 
there would be a share icon next to the information. The other student would need this 
information to complete a step of the problem and would see a listen icon to know 
there was some information they needed to get from their partner. 

To test our experimental hypotheses, two problem sets were created for both the 
collaborative and the individual ITSs. One set focused on procedural knowledge of 
equivalent fractions while the other set focused on conceptual knowledge of equiva¬ 
lent fractions. The procedural problem set has four problem types, with four problems 
each, which focus on finding equivalent fractions or determining whether fractions 
are equivalent, either by finding the common factors and reducing the fraction or by 
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denominator are: 


What makes fractions equivalent? 


1 If you have a fraction, you can make an equivalent 
fraction by: (answer individually and then as a group) 


limes large i 

Each student has an opportunity to 
answer the question individually before 
seeing their partner's answer. 


denominator the same. 

adding the same number to the numerator and 
the denominator. 

multiplying the numerator and denominator by 
the same number. 

multiplying the denominator to make the pieces 
smaller and keeping the numerator the same. 


The students have an opportunity to 
discuss the question before choosing a 
group answer, on which they are tutored. 


Fig. 2. Example of a collaborative conceptual problem: creating equivalent fractions to find the 

pattern in the fractions. 


multiplying the numerator and denominator by the same number (see Figure 1). Each 
of the problem types focused on the steps needed to complete that procedure, without 
addressing conceptual questions about why the procedure works. The conceptual 
problem set also has four different problem types and four problems of each type. 
Two of the problem types provide the students with two stories about whether given 
fractions are equivalent that they need to compare and contrast (one story is correct 
and one story focuses on a misconception) or by providing the students with one story 
that focuses on a misconception that students need to address. The other two problem 
types focus on the definition of equivalent fractions by either having the students 
construct equivalent fractions to find a pattern in the fractions or by having students 
manipulate the denominators and numerators of the fractions independently to see 
how they relate (see Figure 2). For both problem types, students then induce a defini¬ 
tion of what it means for fractions to be equivalent. 


2.2 Experimental Design and Procedure 

To test the hypotheses stated above, we conducted a study with 84 4 th and 5 th grade 
students from two US elementary schools in the same school district. The students 
came from a total of six classrooms. The experiment was a “pull-out” design, where 
the student left their normal instruction during the school day to participate in the 
study. (We did so we could collect eye tracking data, which are not reported here.) 
All students worked with the fractions ITS designed for this study and described 
above. Each teacher paired the students participating in the study based on students 
who would work well together and had similar math abilities. These pairs were then 
randomly assigned to one of four conditions: collaborative conceptual, collaborative 









procedural, individual conceptual, and individual procedural. Twice as many students 
were assigned to the collaborative conditions as to the individual conditions. 

Before participating in the pull-out session, the students had two whole class ses¬ 
sions during which they worked individually with the Fractions Tutor during their 
normal class period (on fractions topics other than equivalence). This allowed the 
students to become acclimated with the tutor before the experiment began. During the 
experiment, the students participated in a 25-minute pretest the morning of their par¬ 
ticipation. Throughout the day, the pairs of students participated in the pull-out ses¬ 
sion. Each such session lasted for one hour where during this time, they received 45 
minutes of instruction dependent on their condition. The next school day, the students 
participated in a 25-minute posttest in the morning. The study spanned a total of four 
weeks. After the end of the study, the students again had two whole class sessions 
where they again worked independently on the Fractions Tutor. 

2.3 Pre and Posttests 

We assessed students’ knowledge at two different times using two equivalent test 
forms in counterbalanced fashion. The tests targeted both conceptual and procedural 
knowledge types. Each test had 11 questions, five procedural and six conceptual. 
Each question either received a 1 when all parts were correct or a 0 otherwise. The 
test items were isomorphic to the items used in the practice problems. 


3 Results 


Table 1. Total correct: means (standard deviation) for conceptual and procedural knowledge at 
pretest, posttest, Min, score is 0, and max, score is 5 for procedural and 6 for conceptual._ 





pretest 

posttest 

Conceptual 

Condition 

Individual 

Condition 

Conceptual Problems 

Procedural Problems 

2.00 (1.63) 

0.46 (0.66) 

2.54 (1.56) 

0.85 (1.21) 

Collaborative 

Condition 

Conceptual Problems 

Procedural Problems 

2.04 (1.32) 

0.50 (0.75) 

2.54 (1.20) 

0.82 (0.82) 


Individual 

Conceptual Problems 

1.50 (0.76) 

1.64 (1.28) 

Procedural 

Condition 

Procedural Problems 

0.50 (0.86) 

0.64(1.08) 

Condition 

Collaborative 

Conceptual Problems 

2.08 (1.67) 

2.58 (1.42) 


Condition 

Procedural Problems 

0.92(1.16) 

0.92 (1.16) 


Because the procedural and conceptual tutor problems were fundamentally different, 
each of these conditions was treated separately and the collaborative and individual 
conditions were not compared across problem types. Three students were excluded 
from the analysis because experimenter error, leaving 81 students. We analyzed the 
data by individual so we could evaluate each student’s learning gain. To test our hy¬ 
pothesis that, on tutor activities targeting conceptual knowledge, students working 



collaboratively have higher learning gains than students working individually, we 
conducted two repeated-measures ANOVAs, one for procedural test items and one for 
conceptual items, with condition (collaborative or individual) as a between-subjects 
factor and test-time (pretest and posttest) as repeated measure. For the conceptual test 
items, there is a significant pre/post difference, F (1, 39) = 4.23, p = .046, no main 
effect of condition, F (1,39) = .002 ,p = .966, and no interaction, F (1, 39) = .006 ,p = 
.940. For the procedural test items, there is a marginal pre/post difference, F (1, 39) = 
4.00, p = .053, no main effect of condition, F (1, 39) = .001, = .976, and no interac¬ 

tion, F (1, 39) = .032, p = .859. There were significant learning gains for both the 
collaborative and individual condition and no difference in gains between conditions. 

To evaluate our hypothesis that students working individually on tutor problems 
targeting procedural knowledge have higher learning gains than students working 
collaboratively, we conducted two repeated-measures ANOVA (for procedural test 
items and conceptual test items, respectively) with condition (collaborative or indi¬ 
vidual) as a between-subjects factor and test-time (pretest and posttest) as repeated 
measure. For the conceptual test items, there is no effect of pre/post, F (1, 38) = 2.10, 
p = .16, a marginal effect of condition, F (1, 38) = 3.44, p = .071 with the collabora¬ 
tive group higher, and no interaction F (1, 38) = .65, p = .426. For the procedural test 
items, there is no effect of pre/post, F (1, 38) = .22, p = .64, no main effect of condi¬ 
tion, F {\, 38) =1.12, p = .297, nor an interaction between condition and pre/post, F 
(1, 38) = .22 ,p = .64. There was no learning gain difference between the collaborative 
and individual conditions. The conditional difference reflects the fact that the students 
in the individual procedural group started lower at pretest and remained lower at post¬ 
test. We also analyzed learning curves derived from the tutor logs for evidence of 
learning during tutor use. Specifically, we looked at the slope coefficient in the AFM 
regression equation (see Figure 3), a standard way of analyzing tutor log data [15]. 
Averaged across knowledge components, the slope was 0.27 for the conceptual condi¬ 
tions and 0.15 for the procedural conditions. For the conceptual conditions, 81% of 
the learning curves has a slope of 0.05 or higher (a rule of thumb threshold value for a 
slope to represent effective learning) and for the procedural conditions, 60% of the 
learning curves had a slope above 0.05. 



Fig. 3. Learning curves for conditions targeting conceptual (left) and procedural (right) 
knowledge. The learning curves are averaged across knowledge components encountered in the 
respective tutor problem sets. The red and blue lines represent the actual and AFM-predicted 
values for the collaborative conditions; the green and yellow lines for the individual conditions. 

We conducted two t-tests (for each procedural/conceptual instructional condition) 
with collaborative/individual as the condition to see if there was a difference in the 
















number of problems each student completed. For the procedural instructional condi¬ 
tion, there is a significant difference, t (38) = 2.65, p = .012, with students working 
collaboratively doing fewer problems than students working individually by about 2.5 
problems. For the conceptual instruction condition, there is a significant difference, t 
(39) =3.61, p = .001, again with students working collaboratively doing fewer prob¬ 
lems than students working individually by about 3.5 problems. 


4 Discussion and Conclusion 

We hypothesized that elementary school students working collaboratively with a tutor 
designed to support collaboration would have learning gains from pretest to posttest. 
The hypothesis was confirmed; the students in the collaborative conceptual condition 
had learning gains comparable to those in the individual conceptual condition. In the 
procedural instructional condition, neither the collaborative nor individual conditions 
saw any learning gains. Thus, collaborative instruction might be as effective for ele¬ 
mentary school students as individual instruction, although it appears to be more suit¬ 
able for activities aimed at acquisition of conceptual knowledge. Collaborative learn¬ 
ing activities may have the added benefit that they help students develop social skills 
and leam to work together. 

While students in the collaborative condition saw fewer problems compared to 
their counterparts in the individual condition, they still had the same learning gains as 
the students in the individual conditions. This is consistent with other findings in 
CSCL [17]. This means that when authoring tutors, if collaborative tutors are used, 
fewer problems need to be developed to facilitate learning. However, we controlled 
for time and if we had controlled for number of problems, students in the individual 
condition may have learned as much as the students in the collaborative condition but 
in less time. 

While we had hypothesized that the individual condition would yield greater learn¬ 
ing gains than the collaborative condition for activities geared towards acquiring pro¬ 
cedural knowledge and that the reverse would hold for activities geared towards ac¬ 
quiring conceptual knowledge, we did not find these differences. We may not have 
found these differences because the instructional period was relatively short. On aver¬ 
age the students in the collaborative conceptual condition completed 7 problems. 
Because the problem types were interleaved and not all knowledge components were 
present in each problem type, the students did not always get to practice each 
knowledge component sufficiently. For the collaborative condition, out of the 16 
knowledge components targeted in the conceptual problems, 9 of the knowledge 
components saw (on average, per student) fewer than 5 opportunities to practice a 
knowledge component. However, the students in the individual condition completed 
12 problems on average and had at least 5 opportunities for all 16 knowledge compo¬ 
nents. By lengthening the practice time with the tutor, such as using the tutor for con¬ 
secutive days in the classroom, the students would have more time with the tutor and 
would get more practice. This would help the students to get more practice with the 
individual knowledge components. 



A second explanation for the fact that the hypothesized differences between the 
conditions were not confirmed may be that the collaborative learning condition was 
more novel and perhaps more demanding for students. Put differently, students may 
need more practice with the instructional method of collaborative learning. Especially 
given that the number of skill opportunities was low, one might expect to see better 
performance on the posttest. Other studies have also shown that the introduction of 
new learning strategies can initially lead to worse learning [19]. These initial perfor¬ 
mance losses may initially mask the success of a new learning strategy. 

The fact that there were no learning gains in the procedural conditions may be due 
to the fact that the procedural problems may have been too difficult for the students. 
We also saw that overall for all conditions, the average number of problems solved 
correctly for the procedural problem types on either the pretest or the posttest was 
below one out of five (see Table 1). The learning curves for the individual knowledge 
components do show signs of learning during the instructional session, with an aver¬ 
aged slope across knowledge components of 0.15, well above the 0.05 threshold. 
Though the learning curves show that students start at an error rate above 50%, they 
also show clear signs of improvement. Because many of the procedural problems are 
multistep, the tests may need to be more fine-tuned to the specific knowledge compo¬ 
nents being learned instead of a cumulative approach of getting the entire problem 
correct. To be able to differentiate between the procedural and conceptual knowledge, 
more work will need to be done to develop and test tutors that can target this 
knowledge. 

The study presented in this paper extends ITSs to include support for collaborative 
learning activities. We have showed that collaborative ITSs are a feasible instruction¬ 
al tool to use with elementary school students, with learning gains equivalent to those 
of students working independently with ITSs. The students in the collaborative condi¬ 
tion also expressed enjoyment in working with a partner to solve problems. To the 
best of our knowledge, our study is the first showing significant learning gains with 
elementary school students working with collaborative ITSs. The use of collaborative 
ITS shows initial promise with elementary school students. 
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